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The Effect of Modeling and Visualization Resources on Student 
Understanding of Physical Hydrology 

Jill A. Marshall , 1,3 Adam J. Castillo , 1 and M. Bayani Cardenas 2 


ABSTRACT 

We investigated the effect of modeling and visualization resources on upper-division, undergraduate and graduate students' 
performance on an open-ended assessment of their understanding of physical hydrology. The students were enrolled in one 
of five sections of a physical hydrology course. In two of the sections, students completed homework problems and projects 
using only Excel (Microsoft, Redmond, WA) or MATLAB (MathWorks, Natick, MA) as modeling resources, and in the other 
three sections, some of the homework exercises were replaced with modeling and visualization activities using the interactive 
modeling software COMSOL Multiphysics (COMSOL, Burlington, MA). Other aspects of the course (instructor, syllabus 
coverage, lectures, and textbook) remained the same throughout the study. We performed a repeated-measures analysis of 
variance, which showed that gains from pretest to posttest were statistically significant overall and were independent of the 
section in which students were enrolled for all but one component on the assessment. In that case, students who did not have 
access to the COMSOL modules marginally outperformed the others, but not to the required level of statistical significance. 
These results were complemented by a qualitative investigation of students' interaction with the modeling software. We 
interviewed a subset of students and assigned codes to themes that arose when we analyzed the resulting transcripts. This 
process allowed us to develop a theory of how students were interacting with the modeling and visualization resources. A 
significant theme was the issue of "scaffolding," or supports, with both positive and negative consequences for students, 
depending on their personal preferences and previous experience. © 2015 National Association of Geoscience Teachers. [DOI: 
10.5408/14-057.1] 
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INTRODUCTION 

Need for Modeling and Visualization in Physical 
Hydrology 

In their call to action, Wagener et al. (2010) described 
the enormous challenges facing hydrologic research and 
education today and the "unprecedented opportunity" to 
use advances in modeling and visualization, which are 
"prerequisites for detecting, interpreting, predicting and 
managing evolving hydrologic systems," (p. 8) to address 
them. Merwade and Ruddell (2012) clearly articulated the 
implications of modeling and visualization in hydrology 
education: 

"Considering the extensive use of authentic data, integrated 
modeling, and geospatial visualization in research applica¬ 
tions and in the professional world, training in these 
approaches is becoming necessary for a successful career in 
hydrology. For this reason alone, it seems reasonable to 
suggest a strategy of supplemmting the traditional hydrology 
curriculum with the latest data and modeling approaches ." 
(Merwade and Waddell, 2012, p. 2398). 

In alignment with these calls for incorporation of data 
modeling and visualization into hydrology instruction 
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(Wagener et al., 2012), faculty at the authors' institution 
have developed a series of COMSOL Multiphysics 3 (COM¬ 
SOL, Burlington, MA; Zimmerman, 2006) models of 
hydrological systems, which permit students to visualize, 
explore, analyze, and predict the consequences of changes to 
a hydrologic system and its inputs. These modules comple¬ 
ment lecture-based instruction and other tools for data 
modeling and manipulation, e.g., spreadsheet software, such 
as Excel (Microsoft, Redmond, WA) and open-ended 
programming, and computational environments such as 
MATLAB (on which COMSOL was originally based; Math- 
Works, Natick, MA). 

During the past five years, the modules have been 
implemented in a colisted, upper-division, undergraduate- 
and graduate-level physical hydrology course. The goals of 
this course are for students to (1) develop a quantitative, 
process-based understanding of hydrologic processes; (2) 
gain experience with different methods in hydrology; and (3) 
enhance their learning, problem-solving, and communica¬ 
tions skills. Developing an "... awareness of the totality of 
interconnected (mainly physical) processes involved in the 
hydrological cycle" (Nash et al., 1990, p. 606) has been 
identified as first among the goals of hydrology education, 
and there is increasing recognition that geoscience education 
requires a quantitative focus (Manduca et al., 2008). The 
hydrology community also acknowledges the need to 
connect this quantitative, theoretical understanding with 
knowledge of the methods and practices within the field 
(Wagener et al, 2012). The contents of the course in this 
study matched quite well to those of the largest subset of 


3 See http://www.comsol.com. 
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hydrology courses, i.e., civil engineering hydrology and 
groundwater hydrology, in a study from several decades ago 
(Groves and Moody, 1992). Thus, the hydrology-specific 
goals of the course aligned with those identified by the 
broader community, and the course cannot be considered 
atypical in that regard. What distinguishes this course is the 
development and addition of the COMSOL Multiphysics 
modeling and visualization modules to the curriculum. The 
homework exercises involving these modules are docu¬ 
mented in the online supplemental materials. 4 The syllabus 
for the course is also included in the online supplemental 
materials. 5 

To test the efficacy of the curriculum modules in meeting 
the stated learning goals, the authors developed an open- 
ended assessment of student understanding of physical 
hydrology (Marshall et al., 2013). The assessment consisted 
of three questions, mirroring course goals. The first asked 
students to describe the important physical processes in 
hydrology and how they affect hydrological systems, the 
second asked students to describe the relevant physical laws 
that govern hydrology and how they relate to hydrological 
processes, and the third asked students how they would 
assess the effects of a drought and urbanization on a local 
spring and to predict what the effects would be in the future. 

Regarding the first question, important components of 
the water cycle are precipitation, runoff, evaporation, and 
transpiration, which are also related to solar radiation and 
soil-and-groundwater flow (or infiltration). In colder areas, 
snowfall and snow melting, sublimation, or evaporation 
would be included. Precipitation is any moisture input from 
the atmosphere to the land, including both rainfall and 
snowfall. Evaporation and transpiration comprise the 
moisture return to the atmosphere, which requires an input 
of energy, coupling the water cycle to the energy cycle. 
Transpiration is moisture uptake by plants from roots and 
released to the atmosphere. Soil-and-groundwater flow is 
the redistribution of water underground, which is due to 
gravity, pressure, and capillary forces. 

Regarding the second question, the relevant physical 
laws that govern hydrology are the conservation laws of 
mass and energy and the conservation of momentum. 
Fundamentally, these are the laws of thermodynamics and 
Newton's second law. There are multiple forms of the 
equation showing Flux = (Resistance or conductance coefficient) 
x Gradient. Examples of this include Darcy's law for 
groundwater flow in saturated porous media or the Richards 
equation for unsaturated flow of moisture through soils, 
equations for mass and energy transfer or evaporation and 
transpiration from soil and water surfaces. Manning's 
equation for open channel flow, Fick's law for solute 
diffusion, and Fourier's law for heat transfer. 

For the third question, to predict the effects of continued 
drought and urbanization, a hydrologist would collect 
historical data, especially during nondrought years and 
before urbanization, then continue to monitor for the same 
information. These data would primarily include rainfall and 
spring discharge and, perhaps, water quality. Based on the 
historical data, either a statistical model or a process-based 
model (e.g., a groundwater flow model) would need to be 


4 The homework exercises can be found online at http://dx.doi.org/10. 
5408/14-057sl. 

5 The syllabus can be found online at http://dx.doi.org/10.5408/14-057s2. 


built, calibrated to the historical data. Urbanization can be 
represented by the amount of impervious cover from maps 
or population data (but that isn't really hydrology). Once a 
model is calibrated, it can be used to analyze what might 
happen under different forcing conditions that are not 
represented in the data, such as extreme and prolonged 
drought or continued growth of the city. 

This article reports results of a 3-y study to compare 
precourse and postcourse responses of students (both 
graduate and upper division undergraduate) in the physical 
hydrology course. Depending on the semester in which they 
took the course, students were tasked with modeling 
hydrologic phenomena using either Excel and/or MATLAB 
alone or with the application of COMSOL Multiphysics as 
part of the assigned homework. During the course of the 
study, the instructor remained the same, and other aspects of 
the course (lecture, exams, student projects, and presenta¬ 
tions, etc.) were deliberately held as constant as possible. 
Examination of course grades and pretest scores indicated no 
systematic variation in the student population over this time; 
however, possible unidentified variation in the population 
constitutes a limitation of the study. 

Specifically, we sought to determine 

• How do different modeling utilities compare in terms 
of enhancing student mastery of course goals as 
assessed by before and after tests? 

• How do students describe their interactions/experi¬ 
ences with different approaches to modeling and 
visualization? 


STUDY DESCRIPTION 

Setting 

The study took place in five sections of a colisted, upper- 
division, undergraduate and graduate, physical hydrology 
class, offered in a department of geological sciences, over the 
course of 3 y. Each of the sections was taught by the same 
instructor, one of the authors (M.B.C.). The class is based on, 
and closely follows, the textbook Physical Hydrology by S. L. 
Dingman (2008), a later edition of the most often used 
textbook cited by Wagener et al. (2007). The semester-long 
class is targeted toward upper-division, undergraduate 
students and new graduate students studying hydrology or 
hydrogeology. The class covered the following broad topics, 
which included all aspects of the hydrologic cycle: (1) 
atmospheric and climate processes; rainfall; (2) snow and 
snowmelt; (3) unsaturated zone and infiltration processes; 
(4) evaporation and transpiration; (5) runoff processes, 
streamflow, and watershed hydrology; and (5) groundwater 
hydrology. Roughly 2 to 3 weeks are devoted to each topic. 
(A sample syllabus is included in the online supplementary 
materials 5 ). 

The topics are presented such that a molecule of water is 
essentially tracked through the hydrologic cycle, beginning 
in the atmosphere, where it condenses to form rainfall or 
snow, then as rain and snow (with snow potentially 
accumulating on the ground), with some water going back 
to the atmosphere (evaporation and transpiration). Infiltra¬ 
tion across the soil surface and soil moisture flow are then 
discussed, with water that did not infiltrate becoming runoff 
on the land surface and eventually forming streamflow. 
Finally, water that infiltrated through the unsaturated soil 
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reaches the water table of aquifers. At this point, ground- 
water hydrology is covered, the last topic of the class. At 
each step, the pertinent physics, and to a certain extent, 
thermodynamics, are taught. 

The class grading is heavily weighted toward assign¬ 
ments, with homework comprising 50% of the final grade. 
One homework assignment is assigned approximately every 
two weeks with a total of six homework assignments 
throughout the duration of the semester. The modeling 
and visualization activities, discussed in detail below, were 
integrated into the third and sixth (final) homework 
assignments (see online supplemental materials 4 ), when 
the instructor felt that modeling and visualization would 
particularly help in the students' learning and appreciation 
of the topic. 

Participants 

In all, 80 participants, out of a total enrollment of 95, 
consented to participate. We were able to match preassess¬ 
ment and postassessment scores for 51 students. Some 
students completed only the pretest (and were not present 
for the posttest or had dropped the course), and others were 
not present at the beginning of class on the first day and 
completed only the posttest or did not label their assess¬ 
ments in such a way that they could be matched. 
Assessments that could not be matched were used for 
summary statistics only. Additionally, 14 students consented 
to be interviewed. Because not every student consented to 
participate and completed both the pretest and posttest and 
the numbers of participants were too small to disaggregate 
the data by gender or student status, there is a limitation on 
the generalizability of the study results. 

The participants were either upper-division, undergrad¬ 
uate or graduate students enrolled in a section of a physical 
hydrology course, cross-listed as undergraduate/graduate 
and offered in the geological sciences, over a 3-y period. The 
course is a requirement for undergraduate students majoring 
in geology with a hydrogeology emphasis or geosystems 
engineering and hydrogeology (a hybrid program between 
petroleum engineering and hydrogeology) and is an elective 
for general geology and environmental science students. 
Calculus and a previous introductory hydrogeology course 
are prerequisites. Graduate students taking the course might 
be from the geosciences or engineering, occasionally other 
areas, and for them, there were no prerequisites, and 
enrollment was based on instructor consent. The difference 
in prerequisites for the undergraduate and graduate versions 
of the course tended to yield overlapping distributions in 
terms of preparation, particularly mathematical preparation, 
for the undergraduate and graduate students. In other 
words, graduate students were not, on average, likely to be 
better prepared technically than undergraduates in terms of 
physical hydrology. 

Modeling and Visualization Resources 

During the first year (two sections of the course), 
students were required to use Excel spreadsheets or 
MATLAB to manipulate data and produce graphs/plots 
through which trends in hydrologic interactions could be 
visualized and analyzed. Arguably, Excel presents only the 
crudest modeling capability, but using it, students might, for 
example, plot the Darcy flux (described below) versus head 
gradient by entering the equation corresponding to Darcy's 
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Law into Excel. They could then identify trends from the 
resulting graph of pressure versus output flow, arguably 
performing rudimentary modeling and analysis. 

During the second 2 y (three sections of the course), 
students were given access to, and required to use, 
COMSOL Multiphysics to model some of the same hydro- 
geological phenomena. COMSOL Multiphysics is a generic, 
finite-element, numerical modeling software. Its origin can 
be traced back to the partial-differential equation numerical 
solver toolbox for MATLAB, which then evolved into 
independent modeling software with a user-friendly, Win¬ 
dows-based, graphical user interface. The COMSOL Multi¬ 
physics user interface integrates all modeling aspects, from 
choosing which governing differential (conservation) equa¬ 
tions to solve, setting boundary conditions and internal 
domain parameters or coefficients, building structured or 
unstructured finite-element meshes, to postprocessing and 
visualization of results, including generation and viewing of 
animations. The workflow from model conceptualization, 
domain and geometry setup, finite-element mesh genera¬ 
tion, solution (or actual model run) to postprocessing is all 
tightly integrated and sequentially arranged. 

The assigned homework problems that use COMSOL 
are included as part of the online supplemental materials. 4 
Figure 1 shows screenshots from a COMSOL Multiphysics 
exercise modeling a Darcy tube and allows students to 
discover Darcy's Law for themselves. The model shown was 
created for students using COMSOL Multiphysics, and 
students were asked to modify the inputs and interrogate the 
results, i.e., a rudimentary sensitivity analysis. Figure 1 also 
illustrates the COMSOL Multiphysics workflow (left side of 
the screenshot); the user would have to go sequentially 
through all the tabs from top to bottom, but in this case, 
these have already been prepopulated. This example shows 
groundwater flow through a tube packed with sand, i.e., a 
Darcy tube. Darcy's law—the fundamental equation de¬ 
scribing fluid flow through porous media—was empirically 
derived through experiments by Henry Darcy with these 
tubes. The students were essentially made to replicate 
Darcy's experiments computationally and digitally using 
COMSOL Multiphysics. In Fig. 1, water is injected from the 
left into the tube and comes out on the right end (arrows 
indicate the flow). The flow is driven by a linear pressure 
drop; the pressure field is indicated by shading in the circular 
cross sections. 

Figures 2 and 3 illustrate two other examples of assigned 
problems that use COMSOL Multiphysics. Figure 2 shows a 
screenshot of a model for unsaturated flow through a two- 
dimensional vertical cross section of soil with a root serving 
as a macropore (i.e., a fast-flow path, or "wormhole"). It 
shows the saturation of the soil (l being saturated) some 
time after infiltration from the top started. This model solves 
the Richards equation, which is a nonlinear, partial- 
differential equation. The students were introduced to the 
equation in class lectures, but such a numerical model 
simulation using the Richards equation is far beyond what 
the class would normally cover. For example, writing their 
own programs to solve this would require many semesters of 
courses in mathematical modeling. The idea of this exercise, 
as with other COMSOL Multiphysics problems, is to 
investigate how the system works when certain parameters 
are changed. The modeling aspects, for the most part, 
remained as a black box for the students, although the 
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FIGURE 1: Screenshot of a three-dimensional COMSOL Multiphysics model of saturated groundwater flow through 
a tube filled with sediment, i.e., a Darcy tube. The model solves the steady-state groundwater flow equation, i.e., the 
Laplace equation, with flow described by Darcy's Law. 


instructor gave a very brief discussion of what the software 
was doing, i.e., that it was numerically solving the governing 
differential equations. 

The third example shown (Fig. 3) models two-dimen¬ 
sional groundwater flow through a regional aquifer where 
flow through the aquifer is driven by sinusoidal pressure (or 
head/water table) variations at the top of the domain or land 
surface. This is a typical, if not classical, configuration for so- 
called topography-driven regional groundwater flow. The 
screenshot (Fig. 3) shows one output for such a model, 
showing the pressure field driving the flow (black flowlines). 
The actual solution or calculation of such a flow field is again 
far beyond the expectation for the students in the class. 
However, they were shown conceptual cartoons of these 
flow fields in class lectures. In this assignment, the students 
interrogated the flow and pressure fields based on varying 
model inputs. 

Despite the capabilities of COMSOL Multiphysics, in 
this study and for the classes we investigated, the modeling 
was kept mostly opaque. The students were taught and 
expected to know the underlying physical processes and 
corresponding equations inherent in the models, but 


minimal modeling background was expected or introduced. 
The COMSOL Multiphysics models were designed to be 
virtual canned, i.e., recipe-driven, experiments. During 
semesters when COMSOL problems were not assigned, 
no directly corresponding questions replaced these prob¬ 
lems, as this would not have been possible. However, 
problems that required hand calculations that could also be 
implemented in Excel or MATLAB were sometimes assigned 
in lieu of the problems involving COMSOL. 

ANALYSIS 

The authors used both quantitative (before and after 
tests) and qualitative (student interviews and observations) 
methods to assess the effectiveness of the various methods 
of implementing modeling. Students in all sections of the 
course were informed about the study on the first class day 
and asked for informed consent to participate in the study, 
following an approved institutional review board protocol. 
Most students gave consent; however, not having every 
student participate may represent a selection bias and is a 
limitation of the study. 
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FIGURE 2: Screenshot of a vertical two-dimensional COMSOL Multiphysics model of unsaturated soil water flow 
due to infiltration from the surface and affected by a root macropore representing a preferential flow path. The model 
solves the transient Richards equation, a nonlinear partial-differential equation. 


Quantitative Assessment of Understanding 

The quantitative design involved two groups that were 
enrolled in different semesters during which different 
modeling resources were used by students (i.e.. Excel/ 
MATLAB or manual calculations versus COMSOL Multi¬ 
physics). The same instructor taught all the classes analyzed 
here, using the same syllabus, lecture files, and format. Class 
assignments that included the COMSOL Multiphysics 
exercises were graded by graduate-student teaching assis¬ 
tants. The same dependent variable (gain score on a physical 
hydrology assessment developed for this study) was 
measured at the beginning and end of each semester. 
Students completed the pre/post assessment, yielding scores 
ranging from 0 (blank or no relevant response), 1 (some 
recognition of concepts, knowledge from precollege curric¬ 
ulum), 2 (basic understanding, college level), and 3 (full 
understanding, what might be expected of an advanced 
hydrology graduate student). Development of the instru¬ 
ment and the rubric is detailed in Marshall et al. (2013). 

Typical responses are described in detail in Marshall et 
al. (2013). In general, responses that scored a 1 on the first 
question—hydrologic processes—mentioned components of 
the water cycle as it might be presented in precollege 


courses, i.e., precipitation and evaporation but not infiltra¬ 
tion or transpiration, and focused on the sun as the driver. 
Reponses that scored a 2 contained a complete list of 
processes, and responses that scored a 3 contained a 
complete list of processes and related them to the presence, 
movement, and storage of water. Typical responses that 
earned a 1 on question 2—laws that govern hydrology— 
listed only the name of a law (often Darcy's law). Responses 
that scored a 2 gave some indication of the conservation 
laws and mentioned that drivers (thermodynamics, gravity) 
and resistive elements determine flow. Responses in the 3 
category gave a clear description of how Darcy's law, 
conservation of mass, and thermodynamics govern the 
processes controlling the storage and flow of water. 
Responses that scored a 1 on the third question—how to 
predict the effect of drought and urbanization—simply 
mentioned comparing present conditions to previous (non¬ 
drought) conditions. Responses garnering a 2 listed appro¬ 
priate measurements and a plan to categorize the present 
and compare the present conditions to input/output trends 
in historical data. Finally, responses earning a 3 described a 
plan to develop a model based on physical laws and 
historical data and to use it to predict future outcomes. 
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FIGURE 3: Screenshot of a two-dimensional COMSOL Multiphysics model of topography-driven regional 
groundwater flow through a rectangular aquifer with the top boundary representing periodic sinusoidal variations 
in pressure. The model solves the steady-state groundwater flow equation, i.e., the Laplace equation, with flow 
described by Darcy's Law. 


Participants' modeling knowledge was assessed indirectly by 
the third item on the preassessment and by general 
awareness of the population enrolling in this course, that 
is, students enrolling in this class are not likely to have an 
extensive knowledge of COMSOL Multiphysics modeling. 
No student mentioned using COMSOL Multiphysics to 
make predictions on the preassessment. 

All responses were blind coded (i.e., coders had no 
knowledge of whether they were assessing a pretest or a 
posttest response, or what modeling resources students had 
experienced in the classroom) by at least four independent 
reviewers who had been trained in using the instrument. An 
initial comparison of the independent coding results yielded 
greater than 90% agreement between coders. All discrep¬ 
ancies were negotiated, and a consensus rating was assigned 
by the entire team. The pre/post scores were matched and 
pretest, posttest, and gain scores (difference between pretest 
and posttest scores) were tabulated in a file using SPSS (IBM 
Statistics for Macintosh, Armonk, NY) software. Each score 
record was assigned a value for the semester enrolled, which 
correlated with modeling resources used, i.e., whether 
students engaged only with Excel/MATLAB/hand calcula¬ 


tions to meet course requirements (n = 22) or were required 
to use COMSOL Multiphysics (n = 29). 

Our goal was to determine whether there were 
significant pre/post differences in the scores on the physical 
hydrology assessment for students with access to different 
modeling resources, that is, whether the modeling resources 
that students used made a difference in their learning, as 
measured by the preassessment and postassessment. 

We compared the pretest and posttest scores between 
the two groups using a one-way analysis of variance 
(ANOVA). This test compares the medians of two 
distributions and calculates the probability that they in fact 
came from the same population, i.e., that the null 
hypothesis (that there was no difference between the two 
groups) is valid. For this purpose, we considered the three 
assessment items to be independent (orthogonal) mea¬ 
sures; that is, familiarity with hydrological process did not 
necessarily correlate with familiarity with the laws of 
physics or the experimental methods of hydrology. There¬ 
fore, we performed a separate ANOVA for each of the three 
assessment items. To minimize the chances that repeated 
testing might result in a Type-I error (that we might reject 
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the null hypothesis even though there was in fact no true 
difference between the groups), we required a more- 
stringent level significance, p < .01, as opposed to the p 
< 0.05 level typically accepted. (Note that a typical 
approach to the issue of repeated testing is to use the 
Bonferroni correction, i.e., to divide the required signifi¬ 
cance level by the number of tests, in this case, three.) 

In addition, because the number of subjects was small 
for the different groups, we were careful to check that our 
data met the requirements for ANOVA. For an ANOVA to 
be valid, (1) there must be no outliers in either group, (2) 
each group's data must be normally distributed, and (3) each 
group must have equal variance (homogeneity of variances). 

To test for outliers, we created box plots in SPSS, from 
which we identified one gain score as an outlier. That score 
resulted from a case in which a student had a clear, well- 
articulated pretest but left portions of the posttest substan¬ 
tially blank, possibly due to time restrictions. This data point 
was eliminated from the set, leaving a total of 50 matched 
pretest and posttest scores for further analysis. 

Because our sample size was large enough, we used 
graphical methods to determine whether the data met the 
normality assumption for ANOVA. The box plots used to 
check for outliers indicated a normal distribution of gain 
scores for each question, once the lone outlier was removed. 
We also created Q-Q plots (plot of expected versus observed 
distribution of scores), and data values appeared to follow 
the 45° normal line; therefore, we judged that the dependent 
variable was normally distributed for both groups. 

Finally, we tested for homogeneity of variances in the 
gain scores because the one-way ANOVA assumes that the 
population variances of the dependent variable are equal for 
all groups. If the variances are unequal, the Type-I error rate 
is affected. There was homogeneity of variances of the gain 
scores for the process assessment component (p = 0.63), the 
law assessment component (0.57), and the methodology 
assessment component (p = 0.47), as assessed by Levene's 
test of homogeneity of variance, meeting that requirement. 

Qualitative Assessment of Interactions With Modeling 
Resources 

In addition to the statistical analysis of student learning, 
a volunteer sample of students were interviewed about their 
experiences in the course, in particular, their interactions 
with whichever modeling and visualization resources they 
had been required to use during the semester. In all, 14 
students were interviewed, representing a sampling of 
undergraduate and graduate students, male and female 
participants, and students enrolled in each of the sections. 
Interviews were recorded and transcribed. In addition, 
students were observed as they interacted with a teaching 
assistant for the course during help/study sessions for the 
course. All the resulting artifacts (interview transcriptions 
and observation notes) were then independently open coded 
for concepts related to modeling and visualization and for 
their interaction with learning and other aspects of the 
course (see, for example, Mann [1993] for an explanation of 
the coding process in grounded analysis). The interviewed 
students presented both positive and negative perspectives 
on different aspects of the course, making it seem unlikely 
that the students who were interviewed, albeit a volunteer 
sample, presented a systematic bias. 
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The concepts identified in the open coding were 
compared, common themes were identified, and common 
terminology and coding schemes were negotiated. Inter¬ 
views were independently coded using this scheme, and a 
subset of the independently assigned codes were compared 
yielding agreement at the 98-99% level. After all interviews 
had been coded, a common categorization scheme for the 
codes was developed by negotiation, and a theory of 
student interaction with the visualization resources was 
developed. 

QUANTITATIVE RESULTS 

As a first step in comparing the results of the posttest to 
those from the pretest, we created before and after 
histogram plots for each of the three components of the 
assessment in each semester. In comparing the frequency 
histograms for scores on each of the three assessment 
questions, a shift toward more positive scores from the 
pretest to posttest on each component of the assessment was 
evident for each semester the course was taught, i.e., both 
for the students who used Excel/MATLAB (year 1) and for 
those who used COMSOL (years 2 and 3). Figure 4 shows 
the before histogram (top row) and after histogram (bottom 
row) for each of the three assessment components for 
students who had access to COMSOL (white bars) and 
those who did not (gray bars). 

To test for the significance of this difference, we 
examined the main effect of time using a repeated-measures 
(within-subjects) ANOVA, which compares the mean of the 
pretest scores to the mean of the posttest scores and 
determines the probability that the pretest and posttest 
results might have come from the same distribution (i.e., 
that there was no change in the scores from the pretest to 
the posttest). Table I shows the results for the process, laws, 
and methodology questions, respectively. Each indicates a 
difference between pretest and posttest scores on all three 
questions that is significant at the p < .001 level, meaning 
that, overall, student scores improved from pretest to 
posttest. There was statistically significant learning in each 
case. 

Table II shows the interaction between the semester a 
student was enrolled in the course, indicative of the 
modeling and visualization resources that were available to 
him or her, and the main effect of time (pre/post) on scores 
for each component of the assessment. There was no 
statistically significant interaction between the semester a 
student participated and the process component [F(l, 48) = 
0.84, p > 0.05] or the physical laws component of the 
assessment [F(l, 48) = 0.03, p > 0.05] meaning that pre-post 
difference did not depend on the semester in which the 
respondent was enrolled, i.e., which modeling resources 
they used, for these two questions. 

The interaction between semester and the methodol¬ 
ogy component of the assessment, however, approached 
significance [F(l, 48) = 6.08, p < 0.05 (but not <0.01)] 
leading us to investigate a possible difference on this 
component between the two groups. Inspection of the 
means plots for the two groups shows that group that used 
Excel, showed a larger improvement (mean = 0.77, SD = 
0.65) on the methodology component of the hydrology 
assessment than the group that used COMSOL Multi¬ 
physics (mean = 0.29, SD = 0.73). Although this result is 
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FIGURE 4: Histograms of scores on the processes, laws, and methodology components of the pretest (top row) and 
posttest (bottom row) for students who used COMSOL Multiphysics as a computing resource (white) and those who 
did not (gray). 


only marginally significant, it indicates that COMSOL 
Multiphysics appears to be no more effective, and possibly 
less effective, in raising scores on this one component of 
the assessment. 

QUALITATIVE RESULTS 

Concepts that arose in the initial independent coding of 
the qualitative data (interviews and observations) included 
time, instructions, visualization, deriving equations, linking 
equations to concepts, fundamental physics, identifying 
trends (in hydrological data), manipulating data, instruc¬ 


tions, expectations, satisfaction, familiarity, software opera¬ 
tion and syntax, acquiring skills and tools (in hydrology and 
hydrology education), and specific topics (soil mechanics, 
snow melt, etc.) and course elements (homework, lecture, 
instructor, COMSOL). Upon reviewing these themes, three 
major categories emerged from the data: affordances, 
barriers, and preferences. Of these, the first two pertained 
directly to students' interaction with modeling resources, 
and the third served to situate the students' experiences. 
Table III documents the categories and the themes that 
emerged under each. 
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TABLE I: ANOVA results showing the main effect of time (pre/post) for the three components of the assessment: hydrological 
processes (process), physical laws (law), and methods of hydrology (method). Note that the effect of time is highly significant (/) < 
.001) in each case. 


Source 

Type-III Sum of Squares 

df 

Mean Square 

F 

Significance 

Partial r\ 2 

Process 







Sphericity assumed 

7.840 

1 

7.840 

22.718 

0.000 

0.317 

Huynh-Feldt 

7.840 

1 

7.840 

22.718 

0.000 

0.317 

Law 







Sphericity assumed 

14.822 

1 

14.822 

50.782 

0.000 

0.509 

Huynh-Feldt 

14.822 

1 

14.822 

50.782 

0.000 

0.509 

Method 







Sphericity assumed 

6.250 

1 

6.250 

23.558 

0.000 

0.325 

Huynh-Feldt 

6.250 

1 

6.250 

23.558 

0.000 

0.325 


Affordances 

Components or characteristics of the modeling resourc¬ 
es (and instruction in general) that students cited as helping 
them learn were characterized as affordances. As noted in 
Table I, visualization was a major concept in the category of 
affordances (all student responses are identified by initials). 

"We're talking like orthographic effect and when you can see 
like a picture of a mountain and then how the air moves up 
it. That really helps like with the visualization and then the 
remembering and understanding.. (A.B.) 

"I like [the COMSOL modides]. They're cool. The first one 
was really interesting to see... It's just — I'm a visual 
learner... It had 3D models for, it was infiltration, and it 
showed the different conductivities of soil and stuff, so it was 
cool. There was a block that wasn't as permeable as the one 
above, so it was interesting." (U.G.) 

Students also noted visualization as an affordance in 
terms of video of actual physical phenomena as well as 
laboratory demonstrations. 

"Even though I read a lot of times beforehand about the 
eddies, the concept, and the papers about eddies, but I really 


cannot understand how, um, the eddies looks like [sic] in 
nature and, uh, he shows some experiments on dye, uh, how 
it's heated and then it propagates, and then I can understand 
better about the concept." (P.L.) 

Students cited opportunities to identify trends, to 
synthesize the "big picture" and analyze real data, as 
affordances that helped them to learn: 

“You coidd find, or like these temperatures, okay, what/how 
does this temperature change with elevation? And then, how 
does this pressure change?" (A.B.) 

"What I learned how to do was put it all together and not 
just schematically, but put it all together with some sort of 
physically derived equation." (A.J.) 

"When you analyze it, it helps you like think aboid it and 
[to] make sure so if you have it right and [to] help it set in, 
like you're understanding in your mind better." (A.B.) 

Students also noted acquiring skills or tools as an 
affordance: 


TABLE II: ANOVA results showing the interaction of the main effect of time (pre/post) with the semester in which students were 
enrolled in the course, indicative of which resources they were required to use. 


Source 

Type III Sum of Squares 

df 

Mean Square 

F 

Significance 

Partial r\ 2 

Process x semester 







Sphericity assumed 

0.291 

1 

0.291 

0.842 

0.363 

0.017 

Huynh-Feldt 

0.291 

1 

0.291 

0.842 

0.363 

0.017 

Law x semester 







Sphericity assumed 

0.008 

1 

0.008 

0.026 

0.872 

0.001 

Huynh-Feldt 

0.008 

1 

0.008 

0.026 

0.872 

0.001 

Method x semester 







Sphericity assumed 

1.461 

1 

1.461 

6.078 

0.017 

0.112 

Huynh-Feldt 

1.461 

1 

1.461 

6.078 

0.017 

0.112 
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TABLE III: Categories and themes arising from qualitative 
analysis of interviews. 


Barriers 

Instructions/examples (lack of) 

Time 

Lack of clear expectations 

Affordances 

Visualization 

Identification of trends 

Synthesis 

Analysis of data 

Relating concepts to equations 

Acquiring tools/skills 

Preferences 

Appreciation/satisfaction 

Difficulties/challenges 

Familiarity 


"We did a lot with MATLAB, and I think it was good 
experience to learn how to use it." (A.B.) 

"I knew nothing about Excel Spreadsheets. Even if I didn't 
get to where I wanted to go, I've come a long way." (P.B.) 

"I think that could be really useful, especially with 
COMSOL being something that would be very important 
to anyone in this room (um) at one stage or another, maybe 
more so to some than like geomorphologists or geographers, 
but for the hydrologists that would be very, very important." 
(A.J.) 

Barriers 

Things that stood in the way of learning, or made 
learning difficult for students, were characterized as barriers. 
This included things that were needed, e.g., instructions, but 
lacking. 

"There were some idiosyncrasies in the [COMSOL] program 
that didn't come across in his walk-through. For instance, if 
you clicked out of tab and clicked back into it, it didn't save 
your changes, and I never could figure out why or if it was 
actually running what I wanted it to. So, things like that... 
trial and error." (S.C.) 


"They're just very lengthy a lot of the time, and it's very easy 
to just do [i.e., put] extra parenthesis or missing one 
parenthesis, and then it's just like the whole thing's wrong." 

(J.p.w.) 


"Whereas with COMSOL, because it was, the whole 
interface was unfamiliar, and the way [the instructors] 


wrote up the instructions for the COMSOL, I found sort of 
difficult to follow ... so I found myself sort of more wrapped 
up in that than in actually interacting with the software and 
thinking about what was going on. ” (A.W.) 

The length of time to complete assignments was also 
given as a barrier to learning: 

"It's definitely going to be different because you have to go 
out and find your own data set, apply concepts that we’re 
learning in the class to a real-world application, and try to 
make paper material out of it, which is going to be helpful; 
it's just going to be time intensive. It's going to be helpful but 
very painful." (N.M.) 


"But, like I said, those first couple of homeworks took me 20- 
plus hours for one homework assignment. It's not like this is 
all I have to do. You know, so it's a little, at times, it's been a 
little frustrating." (M.K.) 

Preferences 

Many concepts that arose in the coding fell into the 
category of preferences, and students' preferences clearly 
moderated their interactions with the modeling resources in 
the course. In the interviews, students uniformly expressed 
appreciation for the efforts of the instructor, regardless of 
whether they felt the course had been a successful 
experience for them, e.g., 

"One thing is that I'm really glad he's [including modeling]. 
The fact that he did that shows that he's real interested in 
teaching and also in the classroom... I really appreciate 
being able to be in a class like that." (P.B.) 

However, students also cited difficulties and challenges, 
often linked to their personal capabilities and skills: 

"Empirical equations always feel a little bit more arbitrary to 
me, and it's more to memorize, and I am not good at 
memorizing equations, so they always feel a little more 
challenging to me in that sense as well." (A.W.) 

"I don't, like, that kind of programming situation is 
something I can do, but I already knew that’s not one of 
my strengths. Like, it's not something that comes very easily 
to me." (M.K.) 


"There's one problem on one of the assignments that did 
require some doing (a little bit) of calculus that did require 
some derivation. That was great, and I think I would be 
willing to do twice (or three times) as much homework if they 
were like that." (D.S.) 

A related theme was familiarity. Students expressed 
comfort (or discomfort) in dealing with familiar (or 
unfamiliar) topics or software: 

"So, I didn't know MATLAB before I had taken this course. I 
understood what it was and what it could do, so that has 
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been kind of a start-up curve, and I don't think that my 
MATLAB codes are very mod, I just think they like get the 
job done." (K.L.) 


" It's just because like I've been using Excel since like middle 
school or whenever you first learn it." (A.B.) 

"Right now we're dealing with stream runoff and that is one 
topic I'm really not familiar with. All the other topics so far 
I've probably been pretty familiar with or I'm doing my 
thesis on so I understand most of the concepts up until now." 
(N.M.) 


DISCUSSION 

How Different Modeling Utilities Compare in 
Enhancing Student Mastery of Course Goals 

Overall, the statistical analysis showed that all imple¬ 
mentations of modeling in this physical hydrology course 
were effective. The difference between pretest and posttest 
assessments was highly significant {p < .001) each semester. 
It was expected, based on previous studies, that COMSOL 
Multiphysics would be effective in enhancing student 
understanding of hydrology. Singha and Loheide (2011) 
reported that 3 h of exploration with a COMSOL Multi¬ 
physics model improved students' estimates of groundwater 
velocities for a sand-and-gravel aquifer compared with their 
estimates made after 4 h of lecture and 1 h of hands-on 
exploration with a physical (sand tank) model. That was a 
small study, involving only eight students in one class; 
however, it indicated that the use of a numerical model, in 
addition to standard lecture and "ant-farm" experiments, 
might enhance student understanding of groundwater rates, 
mechanisms, and the parameters controlling groundwater 
flow and contaminant transport. 

What was unexpected was that both types of modeling 
and visualization resources. Excel and/or MATLAB versus 
COMSOL Multiphysics, would be equally effective. The only 
statistically significant difference in the mean of the 
distributions of gain scores between semesters (those in 
which students were required to model hydrology phenom¬ 
ena using only Excel/MATLAB and those in which they 
employed COMSOL Multiphysics models to study hydro- 
logical systems) was on the methodology portion of the 
assessment, and that was significant only at the p < .05 level. 

This result runs counter to the expectation that the 
manipulation and, in particular, visualization, capabilities of 
the COMSOL model would enable students to achieve 
insights into hydrological phenomena that they would not 
be able to otherwise and problematizes the intuitive call to 
"supplement [...] the traditional hydrology curriculum with 
the latest data and modeling approaches" (Merwade and 
Ruddell, 2012, p. 2398). In this case, the newer tools were 
not shown to be more helpful than older technologies, such 
as Excel (which, as noted in one interview, these students 
had been using since middle school), at least within the 
limitations of this study. 

To investigate the possible reasons for this finding, we 
turned to qualitative analysis of students' experiences with 
the modeling resources as described in interviews. 
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How Students Describe Their Interactions/ 
Experiences With Different Approaches to Modeling 
and Visualization 

In piecing together the major categories arising from the 
qualitative analysis (coding of the interviews), we developed 
a theory of modeling resources as scaffolding. Scaffolding is a 
term commonly used in education to characterize instruc¬ 
tional strategies designed to help students in learning. In its 
broadest sense, all teaching is scaffolding, but the common 
connotation is those temporary supports ("training wheels") 
that teachers implement to move students toward greater 
comprehension and independent problem-solving capabil¬ 
ity. The notion of instructional scaffolding is rooted in the 
assumption that what students can do today with help of a 
more experienced collaborator, they will be able to do 
tomorrow by themselves (Vygotsky, 1978). Here, the actual 
assistance comes from software that encapsulates the 
knowledge and capabilities of more-experienced others, as 
well as from the instructor and teaching assistants. The 
instructors provide scaffolding in the form of access to 
appropriate modeling resources, predeveloped models, 
directions for exploration tasks that will help the students 
develop appropriate skills and conceptual understanding, 
and finally, assistance with the tasks as needed. 

For some students, the scaffolding software clearly 
provided an affordance, i.e., it enabled them to do and see 
things that they would not have been able to do or see 
without it. As noted above, for some students the modeling 
resources facilitated identifying trends over time and 
visualizing relationships. 

"And once I got the graphs in Excel right, I could see what he 
was trying to display in class, and I figured it out for myself 
so that really helped me concentrate on it, learn the 
material." (N.M.) 


",. .and then it's a more visual output there and then, you 
know, running it through there and then saying, you know, 
'Okay, so what are we getting? What effect is this having?"' 
(M.K.) 


"Like, I didn't know anything about COMSOL, but I really 
didn't have to know, because the homework was... it was 
funny 'cause it was very straightforward in many senses. It 
was just like 'click on this, click on that."' (R.F.) 


"And, you know, instead of just being an equation or a line 
graph, you actually got to see in 2D [in COMSOL] what that 
would look like as the contour plot." (S.C.) 

For others, however, the scaffolding actually provided a 
barrier, preventing them from "seeing" the fundamental 
concepts in the same way that a painter's scaffold or 
construction scaffolding might obscure the view of a mural 
or monument, as opposed to providing a perspective or 
access unobtainable without it. Although the scaffolding 
might help in construction or updating of a model, it did not 
help some students in developing an understanding of the 
concept(s) on which it was based: 
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"The reason I would take a physical hydrology course is to 
see things come from first principles, to come from physics, 
and the course material itself is often very encapsulated.... 
We're provided with like Excel spreadsheets and all these 
things, at least for me, the perspective I want out of the class, 
these are all barriers to the fundamental, sort of fluid 
dynamic processes that I'm trying to understand." (D.S.) 

The student quoted above clearly did not see the 
fundamental physics in the outputs of the software. For 
him, the closed-form equations themselves correlated with 
the concepts in a way that that the graphs and images 
produced by the software could not. He would have 
preferred "time and space to engage with the equations 
alone, at a level of rigor that I don't find in class or certainly 
in assignments" (D.S.). 

Another way in which the scaffolding software created 
an unintended barrier for the students was in terms of the 
time and detailed procedures necessary to produce a result. 
Students cited this barrier for each of the forms of software 
used: 

"But for me, because of my not knowing Excel—not knowing 
how to do spreadsheets very well, and not remembering all 
the math and physics—it took me probably twice that 
amount of time, and I just found that excessive." (P.B.) 


"So I think the easiest parts for me on the homework have 
been like conceptual questions, because a lot of the rest has to 
do with figuring stuff out in MATLAB and plotting it in 
MATLAB and things like that." (K.L.) 


"When I was using the COMSOL, I felt like I was just 
struggling with the interface in trying to figure out how I 
need to do what I was suppose to be doing and wondering if I 
was doing the right thing. So, I wasn't thinking much about 
the science that was going on. Urn, it was more about, 'did I 
click the right buttons?" (A.W.) 

The means to remove these two types of barriers may 
run counter to each other. Although the images and displays 
made possible by more-advanced modeling technologies 
might enable novice users to identify trends and synthesize 
concepts, the same technological affordances might hide (or 
at least not highlight) fundamental features of the physics of 
hydrological systems, allowing the user to be unaware of 
"the limitations and difficulties in developing numerical 
models that faithfully represent the system they are 
modeling." 6 

In some cases, even students who benefitted from the 
affordances of whichever modeling resources they used and 
the access those tools provided to solutions for the novice 
modeler, also faced the greatest barriers in using the 
software: 

"Overall, after we can get the Excel to work and then we plot 
whatever we're looking at, that's definitely beneficial, and it 
definitely helps summarize certain processes or helps you 


http://www.comsol.com/books/mmwfem. 


learn the general trend of things, but getting to that point, 
often times, is somewhat difficult." (J.P.W.) 


"That's where the science is... is like modeling and... So I 
think it is really useful; I think the hurdle (maybe) is (like) 
the technical, knowing the program, the start-up associated 
with (like) getting to know a program, but once you 
understand it and if you can actually relate it to physical 
processes, then yeah, I think it's helpful." (K.L.) 

"I got a lot out of [COMSOL modeling], but it was too much 
effort and frustration, and too much time for what I got out 
of it." (P.B.) 

"COMSOL, I kind of wondered at the time why we were 
doing it because most of us have never heard of it and will 
never use it. And so it was a little bit pointless for most of us 
in the class, and it took a lot of time. But the visualization 
was pretty neat." (S.C.) 

The more powerful, versatile, COMSOL Multiphysics 
allowed students to do (and see) more, but was less familiar 
and more challenging. As noted on the COMSOL Web site: 

"Finite element methods for approximating partial differen¬ 
tial equations that arise in science and engineering analysis 
find widespread application. Numerical analysis tools make 
the solutions of coupled physics, mechanics, chemistry, and 
even biology accessible to the novice modeler. ... But with 
this modeling power comes great opportunities and great 
perils ." (emphasis added) (http://www.comsol.com/ 
books/mmwfem). 

Thus, despite the affordances provided, all three 
modeling resources explored in the course of this study also 
created barriers to learning, at least for some students. In the 
end, personal preferences and familiarity had the dominant 
role in determining the balance between the two effects. 

CONCLUSIONS AND IMPLICATIONS 

Although our quantitative results did not show a clear 
advantage for the incorporation of the COMSOL Multi¬ 
physics, the qualitative results indicate the value that 
students placed on the opportunity to learn to use this tool: 

"I think that could be really usefid, especially with 
COMSOL being something that would be very important 
to anyone in this room.. .for the hydrologists, that would be 
very, very important." (A.J.) 

Even students who thought they would never use 
COMSOL in the course of their careers saw value in the 
ability to visualize phenomena, and our program remains 
committed to continuing to offer and expand activities with 
COMSOL Multiphysics as part of the instruction in physical 
hydrology and to make these tools available to the larger 
community. For more information about access to the 
modules, please contact one of the authors. 
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However, these results also indicate the need both to 
make the COMSOL Multiphysics modules more accessible 
and to ensure that they are used in such a way that students 
are "thinking... about the science that [is] going on" rather 
than thinking "did I click the right buttons" to justify the 
expenditure of effort on the part of students and instructors 
to include them in the curriculum. Perhaps not surprisingly, 
these results reiterate that the effectiveness of any tool will 
depend on the needs and abilities of the user, and how she 
or he uses that tool. A newer, more powerful technology will 
not necessarily provide a panacea. Developers need to be 
cognizant of the need to provide self-explanatory and 
intuitive interfaces, with entry points for more-novice users 
(training wheels), which still allow the user to inspect and 
consider the constraints and boundary conditions that might 
have been implemented on his or her behalf. A major step in 
this direction has been a new COMSOL release, called 
COMSOL Server, in which a user (in this case, the instructor) 
can create a model and "freeze" it for others to use, placing 
limitations on the changes that can be made to variables. 
This feature will be employed in future instantiations of the 
course to make the COMSOL manipulations more trans¬ 
parent to the students. Demonstrating that students learn 
more (or more efficiently) with the COMSOL modules as 
they exist would have been in some ways a more welcome 
result; however, solid evidence that more work needs to 
done to make the modules accessible to students is also of 
value. 

At the same time, in giving students access to the latest 
data and modeling approaches, care needs to be taken that 
scaffolding provided does not obscure the physics, with 
modeling tasks supplemented by reflection, closing the loop 
back to the fundamental laws. No amount of modeling and 
visualization will substitute for thinking and articulating 
what the models are telling us. Therefore, open-ended, 
conceptual questions requiring students to describe the 
meaning of the simulation results will continue to be part of 
COMSOL-based homework exercises (see online supple¬ 
ment 4 ). The ultimate goal is for students to "put it all 
together and not just schematically, but .. .with some sort of 
physically derived equation," as A.J. did. 

Finally, this study took place in only one course at one 
institution. Not all students consented to participate, and not 
all participants completed both the pretest and the posttest 
assessment, limiting the generalizability of the results. 
Further, participant numbers were not sufficient to be 
disaggregated by student status (undergraduate versus 
graduate), gender, or, perhaps more important, by the 
student's intended career path. Thus, another outcome of 
this study is the identification of the need for further research 
into the effectiveness of the COMSOL Multiphysics curric¬ 
ulum intervention described here with larger populations. 
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